On Optimal Data Split for Generalization Estimation and Model Selection

Modeling with flexible models, such as neural networks, requires careful control of the model complexity and generalization ability of the resulting model. Whereas general asymptotic estimators of generalization ability have been developed over recent years (e.g., [9]), it is widely acknowledged that in most modeling scenarios there isn't sufficient data available to reliably use these estimators for assessing generalization, or select/optimize models. As a consequence, one resorts to resampling techniques like cross-validation [3, 8, 141, jackknife or bootstrap [2]. In this paper, we address a crucial problem of cross-validation estimators: how to split the data into various sets. The set V of all available data is usually split into two parts: the design set & and the test set F. The test set is exclusively reserved to a final assessment of the model which has been designed on & (using e.g., optimization and model selection). This usually requires that the design set in turn is split in two parts: training set 7 and validation set V . The objective of the design/test split is to both obtain a model with high generalization ability and to assess the generalization error reliably. The second split is the training/validation split of the design set. Model parameters are trained on the training data, while the validation set provides an estimator of generalization error used to e.g., choose between alternative models or optimize additional (hyper) parameters such as regularization or robustness parameters [lo, 121. The aim is to select the split so that the generalization ability of the resulting model is as high as possible. This paper is concerned with studying the very different behavior of the two data splits using hold-out cross-validation, K-fold cross-validation [3, 141 and randomized permutation cross-validation' [l], [13, p. 3091. First we describe the theoretical basics of various cross-validation techniques with the purpose of reliably estimating the generalization error and optimizing the